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Under federal law, “Evidence-based” 
means: 


Tier 1: Strong evidence from at least 1 well-designed 
and well-implemented experimental study. 


Tier 2: Moderate evidence from at least 1 well- 
designed and well-implemented quasi-experimental 
study. 


Tier 3: Promising evidence from at least 1 well- 
designed and well-implemented correlational study 
with statistical controls for selection bias. 


Tier 4: Demonstrates a rationale based on high- 
quality research findings or positive evaluation of 
likelihood to improve student outcomes or other 
relevant outcomes; and includes ongoing efforts to 
examine effects. 


Source: Every Student Succeeds Act, Section 8002 


Why this Topic 


Encouraging schools to use strategies backed by 
research is a key driver of improvement. This notion 
has been embedded in federal education law since 
2002, but most clearly in the Every Student Succeeds 
Act (ESSA). Passed in 2015, ESSA requires each state’s 
lowest-performing schools to implement “evidence- 
based” improvement strategies, but gives states and 
districts flexibility in how the rule is applied.i 
According to ESSA, “evidence” can be as stringent as 
multiple scientifically rigorous studies showing that 
the strategy is effective, or as simple as a research- 
based rationale for why the strategy will likely 
improve student outcomes. 


Given this flexibility, it is important to understand 
how states and districts are actually supporting 
schools’ use of evidence. It may signal the extent of 
likely improvement in the future and help education 


The Every Student Succeeds Act encourages educators 
to use school improvement strategies backed by 
rigorous research. Recent national surveys suggest 
that states and districts share that goal but may often 
be relying on research at lower evidence tiers. 


officials learn from each other’s approaches. 
Information on evidence use comes from a recent 
study of ESSA implementation, completed in 2018. 
This was a particularly important year because even 
though ESSA was passed in 2015, not all states 
received approval from the U.S. Department of 
Education to implement their plans until 2018. 
Therefore, the supports described in this snapshot 
mostly indicate where evidence use under ESSA is 
starting from.i' A later survey and report will examine 
how support for evidence use has changed. 


Data and Analysis 


The data come from two surveys administered in 
2018, each containing a few questions on evidence use 
in school improvement. ii The first survey was sent to 
50 states and the District of Columbia, all of whom 
responded.‘’ The second survey was sent to a 
nationally representative set of 713 school districts, of 
1 


which 96 percent responded. Only districts with at 
least one lowest-performing Title I school received the 
evidence-use questions.” Responses to survey 


Key Findings 


Most states pointed districts and schools to 
evidence on improvement strategies but few 
required schools to choose based on it. 


e Almost all states reported promoting the use of 
evidence-based strategies in their lowest- 
performing schools. Overall in 2018, 49 states 
reported using at least one of a number of 
approaches to promote the use of evidence-based 
improvement strategies (Figure 1). 


questions were tallied across all responding states and 
districts to provide a national picture. 


to the U.S. Department of Education’s What Works 
Clearinghouse, Evidence for ESSA, and other 
organizations that rate the evidence behind 
different educational programs, practices, or 
approaches (43 states). 


e Most states also promoted the use of evidence 


through grant funding criteria. For example, 
districts applying to their state for school 
improvement funds were required to provide (or 


received bonus points for providing) research 


¢ The most popular approach was to provide evidence to support their proposed strategies (34 
information on evidence-based strategies. States states), or a plan to evaluate the strategies as they 
gave this information either directly to districts and carried them out (32 states). 


schools (42 states) or indirectly, by referring them 


Figure 1. State approaches to promote the use of evidence-based strategies in lowest-performing schools 
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Notes: 2017-18 survey of 48 states and the District of Columbia. The lowest-performing schools in 2017-18 include all such Title I schools, whether they were 
identified under the state’s own accountability system, ESSA’s new comprehensive support and improvement system, or previously identified under an 
older federal accountability system (NCLB’s restructuring or corrective action schools, ESEA Flex’s priority schools, 1003(g) School Improvement Grants). 
Oregon and Wyoming were not asked to provide this information because they had not yet identified lowest-performing schools under ESSA when the 
survey was administered, and did not require previously-identified lowest-performing schools to implement improvement strategies in 2017-18. 


e Just over a third of states connected districts and 
schools to federal technical assistance (TA) 
providers. Eighteen states reported making 
referrals to the U.S. Department of Education’s 
Regional Educational Labs and Comprehensive 
Centers, which provide services at the state and 
local levels to increase research and evidence use, 
and to build capacity to improve educational 
outcomes for students. Fourteen states also 
reported sharing information on other technical 
assistance providers who could help with 
implementing improvement strategies. 


e Few states required schools to select strategies 
from a list backed by research, but the vast 
majority of districts reported that their schools 
did so anyway. Just nine states took the potentially 
more stringent approach of requiring their lowest- 
performing schools to select from a list of evidence- 
based improvement strategies identified by the 
state (Figure 1). Still, 90 percent of districts reported 


that all of their lowest-performing schools were 
implementing strategies selected from such a state 
list (Appendix Figure 1). 


Most districts reported that evidence of 
effectiveness was “very important” when 
choosing improvement strategies. 


In 2018, 89 percent of districts felt that evidence of a 
strategy’s effectiveness at improving student 
outcomes was a “very important” consideration when 
selecting strategies for their lowest-performing 
schools (Figure 2).v' By comparison, six other common 
factors, including a school’s grade level, 
district/school capacity, high-priority student 
subgroups, school staffs interest, cost, and 
parent/community input were rated “very important” 
by 88 percent, 85 percent, 77 percent, 71 percent, 

59 percent, and 47 percent of districts, respectively. 


Figure 2. Factors districts consider to be "very important" when selecting strategies in lowest-performing 


schools 
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Grade level of school (i.e., elementary, middle, or secondary) 


District/school capacity to implement the strategies 


Student subgroups in need of improvement 
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Cost of strategies and amount of funding available 


Parent/community input 
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Percent of districts reporting that this 
consideration was “very important” 


Notes: 2017-18 survey of 184 (unweighted) or 3,261 (weighted) districts with lowest-performing schools, which include all such Title I schools, whether they 
were identified under the state’s own accountability system, ESSA’s new comprehensive support and improvement system, or previously identified under 
an older federal accountability system (NCLB’s restructuring or corrective action schools, ESEA Flex’s priority schools, 1003(g) School Improvement Grants). 


But the evidence districts relied on probably 
varies in quality. 


State guidance was one of the most frequent 
sources of improvement strategies. In 2018, 

70 percent of districts reported receiving guidance 
from their state or state-funded TA centers. More 
than half of districts reported consulting state- 
approved lists of vendors, who provide materials, 
conduct trainings, or otherwise support 
implementation of a strategy (Figure 3).vi The 
extent to which the vendors were backed by 
research that could meet ESSA’s evidence 
thresholds is unknown. “ii 


Other common information sources reported by a 
majority of districts include colleagues and 
vendors. In 2018, districts reported getting 
information from colleagues in other districts 

(71 percent), professional associations (57 percent), 


and developers/vendors (56 percent). While 
districts likely consulted these sources for a variety 
of purposes, some sources—such as 
recommendations from colleagues, developers, and 
vendors—are unlikely to be sources for strong 
evidence on improvement strategies. * About half of 
the districts also reported consulting with 
universities (51 percent) and district research offices 
(50 percent). 


Fewer districts reported getting information from 
sources designed to rate and share evidence, such 
as the What Works Clearinghouse and Regional 
Educational Labs. When gathering information to 
select improvement strategies in 2018, 43 percent of 
districts reported consulting the What Works 
Clearinghouse, Evidence for ESSA, or other 
organizations that rate evidence, while just 

16 percent consulted the Regional Educational Labs 
or Comprehensive Centers. 


Figure 3. Where districts get information to help select strategies in lowest-performing schools 
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Notes: 2017-18 survey of 184 (unweighted) or 3,261 (weighted) districts with lowest-performing schools, which include all such Title I schools, whether they 
were identified under the state’s own accountability system, ESSA’s new comprehensive support and improvement system, or previously identified under 
an older federal accountability system (NCLB’s restructuring or corrective action schools, ESEA Flex’s priority schools, 1003(g) School Improvement Grants). 


Appendix 


Data Collection Procedures 


The data presented in this snapshot come from state 
and district surveys administered in 2017-18 (between 
April and September 2018) as part of the Study of 
Implementation of Title I/II] Program Initiatives. A 
similar version of the surveys was administered in 
2013-14. See Appendix B (pp. B-5 to B-8 and 

pp. B-14 to B-16) of a report based on those surveys for 
amore complete description of the sampling and 
weighting approach used in this study. 


State Survey 


The state survey, administered using an electronic, 
fillable PDF form, was sent to the chief state school 
officer in each of the 50 states and the District of 
Columbia. The expectation was that different sections 
of the survey would be filled out by different state 
educational agency staff members with the most 
direct knowledge. The survey had a 100 percent 
response rate. 


The state survey questions related to use of evidence 
asked about the state’s lowest-performing Title I 
schools in 2017-18. These schools could have included 
previously-identified priority schools, as defined 
under the flexibility that most states began receiving 
from the U.S. Department of Education in 2012. This 
flexibility waived certain requirements under No Child 
Left Behind (NCLB), which was the version of the 
Elementary and Secondary Education Act (ESEA) that 
preceded ESSA. A state’s lowest-performing schools in 
2017-18 could also have included previously-identified 
schools in restructuring or corrective action (as 
defined under NCLB), schools receiving School 
Improvement Grants, schools identified for 
comprehensive support and improvement (as defined 
under ESSA), or schools identified as lowest- 
performing under a state accountability system 
distinct from ESSA. 


The state survey questions related to use of evidence 
were not applicable to Oregon and Wyoming. These 
two states had not yet identified lowest-performing 
Title I schools under ESSA when the survey was 


administered and did not require previously-identified 
lowest-performing schools to implement 
improvement strategies in 2017-18. Thus, the survey 
directed them to skip these particular questions, and 
the state survey data presented in this snapshot are 
ultimately based on 48 states and the District of 
Columbia. 


District Survey 


The district survey was based on a nationally 
representative sample of 722 school districts, 
including 545 local education agencies (LEAs, typically 
school districts) drawn from the 2011-12 National 
Center for Education Statistics’ Common Core of Data 
(CCD) and 177 charter LEAs drawn from the 2016-17 
CCD. The same 545 LEAs were used in the 2013-14 and 
2017-18 data collection for the Study of 
Implementation of Title I/II Program Initiatives. The 
177 charter LEAs were added to the 2017-18 data 
collection to more fully capture the experiences of 
charter districts and schools. The district sample was 
drawn using “minimax” random sampling, which is 
designed to equalize the efficiency for two types of 
district estimates: unit-based estimates and 
enrollment-based estimates. The sample design 
considered districts’ poverty status, size, geography 
(Census region, state), and urbanicity. 


Nine sampled charter LEAs were not ultimately 
eligible for the survey. The web-based survey was sent 
to superintendents or their designees in the 713 school 
districts that remained eligible, and 683 surveys were 
returned for a response rate of 96 percent. Applying 
weights to account for oversampling certain 
subgroups and non-response ensures that the final 
district statistics reported are nationally 
representative. 


The district survey questions related to use of 
evidence were only asked of districts who indicated 
that they had at least one lowest-performing Title I 
school. Ultimately, 184 (3,261 weighted) districts 
responded to these questions, which is 27 percent 
(19 percent weighted) of the study’s final sample of 
responding districts. More detail on data collection 


procedures can be found at 
https://www.regulations.gov/document?D=ED-2017- 


ICCD-0124-0006. 


Survey Content 


The surveys pertained to states’ and districts’ 
activities during the 2017-18 school year and 
contained questions related to core areas in ESEA 
Title I and Title I-A: state content standards; 
assessments; school accountability and turning 


Relevant State Survey Questions 


around low-performing schools; and improving 
educator effectiveness. The surveys also contain 
questions related to school choice. The state survey 
focused on state policies and supports provided to 
districts. The district survey focused on the 
implementation of state policies, adoption of district 
policies, and supports provided to schools. The full 
surveys can be found at 
https://www.regulations.gov/document?D=ED-2017- 
ICCD-0124-0007. The specific survey questions that 
were used in this snapshot are reprinted below. 


How is your state promoting the use of evidence-based models, interventions, or strategies by lowest-performing 


Title I schools implementing interventions during this school year (2017-18)? [Question 1-36] 


Approach 


YES NO NA 


a. The state provided district and school leaders with information about evidence-based 
models, interventions, or strategies to improve student performance..............sceeceeeee 1 O 


b. The state provided a list of vetted partners that district and school leaders could engage 


to implement approved evidence-based strategies ................... 


c. The state linked district and school leaders with staff from the U.S. Department of 
Education’s Regional Educational Laboratories or Comprehensive Centers to obtain 
information on evidence-based models, interventions, or strategies to improve student 


DETLOLIMAN CE terseen eerrenteeersrrec crane e tee sess te nee recent eee 


d. The state referred district and school leaders to the What Works Clearinghouse, 


Evidence for ESSA, or other organization that rates evidence to obtain information on 
evidence-based models, interventions, or strategies to improve student performance.... 


District applications for school improvement funds must describe the evidence base for 
proposed interventions, or they receive competitive preference for describing such 
VICE CON eee coe ae ec cetaceans e ee neene veut ane suse atseevans ss tuacpus Suvectetaae recwateceatiecersaterer ieneeeuseacdeierTea 


District applications for school improvement funds must include plans for evaluating 
the effectiveness of interventions, or they receive competitive preference for evaluation 


(Specify): 


NA 


NA 


Thinking about the state’s lowest-performing Title I schools that were implementing interventions during this 
school year (2017-18), what interventions, if any, did the state require during this school year (2017-18)? 
[Question 1-32] 

NOTE: Please select “required” if the state requires the intervention for some or all lowest-performing schools in 
2017-18 


NOT 
Interventions REQUIRED REQUIRED 
h. Schools must implement interventions selected from a list of evidence-based programs and 
FUTOLGLENIS ko Kena at ere Loy? TWINS SETI occccoocacooxaccososocoodaccoses6c09a90NE0L00000000°. 30900 TE IEICE ATE IGHOROECD 1 0 


Relevant District Survey Questions 


How important were each of the following considerations when selecting the interventions to implement in 
lowest-performing Title I schools? [Question 1-15] 


NOT SOMEWHAT VERY DON’T 
Considerations IMPORTANT | IMPORTANT | IMPORTANT | KNOW NA 
a. School staff’s interest in specific interventions..................... 1 D 3 DK 
b. Parent and/or community input... eee eeeeeeeeeeeeeeeeeeee 1 2 3 DK 


c. Grade level of the school (i.e., elementary, middle, or 


SECONGALY) ete tacseaesentrestsuseenses ce svacsusesesesuacsasesetuarecsessess sets 1 2} 3 DK 
d. Student subgroups needing intervention to improve 

ACHISVEMEME: <.2e ade cececede sade censvinevevivevevsuehovestversesdsndedsieteiaats 1 2 3 DK 
e. Cost of interventions and amount of funding available......... 1 2; 3 DK 


f. District and/or school capacity to implement the 
MMECT VENUS v5222522c2:5155ck2cextsesaassgauecsdeenesshe bests oot ssstadocsecesends 1 2 3 DK 


g. Research evidence showing that the interventions were 


effective at improving student OUTCOMES .............s:ceceeeeeeee 1 2 3 DK 
Ti..  SOMEtHING CISC -.. des cee cenectoncscseves senscusestecdesetesenceessnessverenenses 1 2 3 DK 
(Specify): 


Which of the following sources of information were consulted when selecting the interventions to implement in 
lowest-performing Title I schools? [Question 1-14] 


DON’T 

Sources YES NO KNOW 
a. Guidance or advice from the state education department or a technical assistance center 

funded by: the: State vc. .2ccc.cccceccs cc ceusececesssucesesssecussosssas sssusvesocesssuscssunustessvsrsesescesessstecwesssecessads 1 0 DK 
b. A list of vendors approved by the state ...........eeesseesseeeeesseceeeseeeeeseeeceeseeeseeeeeeeseeeeeeaeeeeeaae 1 oO DK 
c. Information provided by the intervention’s developer or Vendo ................:sceeeeeeeeeeeeeeeeeee 1 oO DK 
d. Recommendations from colleagues in other school districts................sscesseeeeeeeseeeeeseeeeeeeeee 1 O DK 
e. Information from a U.S. Department of Education Comprehensive Center ................::ecee+ 1 (6) DK 
f. Information from a U.S. Department of Education Regional Educational Laboratory............ 1 0) DK 
g. Information from the What Works Clearinghouse, Evidence for ESSA, or other organization 

thabrates Vid ence eceteee Oe cere ee ere cas eas ica an sada eu SRLS Saar es tet Tevee res Ore teeeeerTs Oaettieee 1 0 DK 
h. Information from the district’s research/evaluation Off1Ce ............eeseeeeeeeeeeeeeeeeeeeeteeseeereeseee 1 O DK 
i. Information from professional associations ...............::cccssscccessececeeseceseeseeeceeseeeceeseeesesaeeeeaee 1 (6) DK 
j. Information from a college/umiversity researcher ............:ccessecseeseceeeeseeeeeseeecesseeeseeseeeeeaees 1 0 DK 
Ike, BSOMLE | OLNETSOUNCE era ree cetaes ae cceen ere oe a Na I, GALE SOLE gene US SA ya eaui crite et gee eee a 1 0) DK 

(Specify): 

During this school year (2017-18), are all, some, or no lowest-performing Title I schools in your district 
implementing the following interventions? [Question 1-5] 

Interventions ALL SOME NONE 
f. Schools are implementing interventions selected from a list of evidence-based programs 

and modelslidentified by, the Stat w.c.-<c.c-cccce-ccceescocsee-secencsecesseac-sosccsesssucecascsecessanceeasecsseses 2 1 (0) 


Appendix Figure 1. Percent of districts whose lowest-performing schools chose interventions from a state list 
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Notes: 2017-18 survey of 184 (unweighted) or 3,261 (weighted) districts with lowest-performing schools, which include all such Title I schools, whether they 
were identified under the state’s own accountability system, ESSA’s new comprehensive support and improvement system, or previously identified under 
an older federal accountability system (NCLB’s restructuring or corrective action schools, ESEA Flex’s priority schools, 1003(g) School Improvement Grants). 
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i For instance, states were allowed to “freeze” the status of their previously-identified lowest-performing schools for the 2017-18 school year. 
These schools retained their “lowest-performing” designation for 2017-18 even though they may have been identified through an older 
accountability system. As a result, the data in this snapshot should not be interpreted as measuring how faithfully states implemented ESSA’s 
new evidence requirements in 2017-18 because not all of the low-performing schools would have necessarily been subject to these 
requirements. 

ii The surveys were administered as part of the National Center for Education Evaluation’s Study of Implementation of Title I/II Program 
Initiatives, which is more generally documenting the implementation of the Elementary and Secondary Education Act’s Title I and Title II-A 
programs over time. Additional details about the sampling methods and survey instruments can be found in the appendix and at 
https://www.regulations.gov/document?D=ED-2017-ICCD-0124-0006. 

However, only states who met at least one of the following conditions were asked to answer the specific survey questions summarized in 
this snapshot: (1) had already identified lowest-performing schools under ESSA when the survey was administered; (2) required previously- 
identified lowest-performing schools to implement improvement strategies in 2017-18. Two states (Oregon and Wyoming) did not meet either 
of these conditions. The data reported in this snapshot are thus from 48 states and the District of Columbia. 

’ These districts amounted to approximately one-quarter of the total sample, or 184 responding districts. 

i This finding is consistent with recent case studies on evidence use in districts: Rentner, D. S. & Kober, N. (2019). District Leader Interviews: 
How School Districts are Responding to ESSA’s Evidence Requirements for School Improvement. Report from Center on Education Policy: 
https://www.cep-dc.org/displayDocument.cfm?DocumentID=1509. 

vl Only 14 states provided a list of vetted partners that districts and schools could engage to implement approved evidence-based strategies 
(Figure 1), and only half of districts (51 percent) consulted a list of vendors approved by the state when selecting improvement strategies for 
their lowest-performing schools (Figure 3). Yet, the vast majority of districts—90 percent—reported that all of their lowest-performing schools 
were implementing strategies from a state-identified list of evidence-based strategies. This apparent puzzle could be an artifact of differences 
in how the survey questions were worded, or a “social desirability bias” for districts to overwhelmingly say that their schools selected 
strategies from a state list. It could also indicate the presence of different state lists. Perhaps districts interpreted states’ efforts to direct them 
to resources from organizations such as the What Works Clearinghouse or state-provided information about evidence-based strategies as 
“state lists.” 

“il Recent data from a different study of 13 state lists with a total of 151 school-turnaround vendors found that only about half of the vendors 
were backed by research that could meet ESSA’s minimum threshold of “demonstrates a rationale” (Tier 4), and that just 11 percent had 
evidence of effectiveness for improving student outcomes that meet the “moderate” and “strong” evidence levels under ESSA (Tiers 1 and 2): 
Meyers, C. V. & Vangronigen, B. A. (2018). So Many Educational Service Providers, So Little Evidence. American Journal of Education. 125, 
109-139. 

‘x Rentner, D. S. & Kober, N. (2019). District Leader Interviews: How School Districts are Responding to ESSA’s Evidence Requirements for School 
Improvement. Report from Center on Education Policy: https://www.cep-dc.org/. 

Rentner, D. S., Kober, N., & Braun, M. (2019). State Leader Interviews: How States are Responding to ESSA’s Evidence Requirements for School 
Improvement. Report from Center on Education Policy: https://www.cep-dc.org/. 

Meyers, C. V. & Vangronigen, B. A. (2018). So Many Educational Service Providers, So Little Evidence. American Journal of Education. 125, 
109-139. 


